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ABSTRACT , . '] _ | .* ^ , 

'■ , A review of Project Themis, which focused on .the 
computer as a teacher, is presented. Descriptions of research into 
four related areas of strategy are first presided. These research^ 
areas are: 1) leairner strategies focusing on the internal cognitive 
and personality processes of the learner who is involved in -/ 
computer-based learning tasks; 2) training strategies sue If as 
computer-managed instruction, adaptive testing, ^adaptive 
instructional systems, the automation of intellectual and personality 
measures*, and the development of sequential and tailored testings 3) 
validation strategies, concerning problems, arid research heeds of 
.ongoing Nav^y training programs; and 4) compter system strategies, 
including management systems, interactive systems and data analysis * 
development* Twk development of hutfan resources through the .Project 
are documented^ and the principal findings are presented. <PB) 
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- ABSTRACT 



At the inception of the ONR Themis Project in 1967, re- ^ 
searchers were concerned with "the computer as the teacher." 
At the end of five years of research, the"* principal finding 
of the FSU investigators is that the context of computer- 
based training is broader than the original conception , . and 
* Pp°fits best from a management model for instruction . .During 

research period7~ the four interrelated areas 6f investi- 
gation were learner strategies, training strategies, valida- 
tion strategies, and computer systems strategies. The 
primary focus of investigation for the learnerxstrategy 
research was the study of internal cognitive and persbnality 
processes of the learner while in computer-based learning 
tasks. The trend in all learner strategy research is 
toward exploration of ajti adaptive instructional desigrv.that 
interacts x^ith task and) learner characteristics. The area 
erf computer-managed instruction, first priority in the area 
of training strategies , was* investigated with positive findings* 
on its training effectiveness and cost benefit outcomes. Both 
adaptive testing and adaptive instructional systems were 
researched as training strategies, as were the automation of 
irttellectual and personality measures, and the development of 
sequential and tailored testing. Applications of simulation and 
information retrieval systems in training and the use of simula- 
tion for research indicated that the techniques should be imple- 
mented and investigated in a wider array of applications. In 
the area of validation strategies during the Themis Project, 
every effort was made to interrelate with ongoing Navy training, 
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s ^associated problems, and its need for research. In computer 
ystfem strategies, three areas of study and development w^re « 
arried on during the five years of Themis at FSU: management 
\ vy^ystems, interactive systems, and data analysis development. 
Following the descriptions of the^e four strategy areas, 
the' final report ^documents the results of tfte development of 
human resources through the Themis Proje'dtfT and lists the varied 
technical documents which were . produced' ^s a result of project- 
related activities'.^ *\ ^ v * * * 
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1.0 The FSU Themis Project v a 

■* If the group of researchers who. began the ONR Themis Project 
in 1967 had envisioned the contents of the final report, the image 
of the table of contents v/ould have centered about "the computer 
as the teacher." °The group mind set was framed about the vision of 
enriched , detailed, student-computer interaction with training con- 
trol via optimization. Now, in the light of the five-year research 
pathway, the FSU investigators can report their principal findings: 
the context of cofriputer-based training is broader than the original 
conception and profits best from a management model for instruction. 
The FSU team found that there is ^no lingular way to individualize 
instruction to its optimal level without employing some of the 
older techniques, such as group discussion, or the newest ones, such 
as multimedia splitscreen presentations. This requirement for a 
broad multifaceted approach to training^resulted from facing research 
a$ it evolved through study after study toward "the computer as the 
manager of instruction." Using a computer-managed instructional 
(CMI) model to encompass computer-assisted instruction (CAI) , simu- 
lation, adaptive testing, natural language dialogues, media manage- 
ment, scheduling, record keeping, and evaluation, the potential of 
each of these components as training processes became most? enhanced. 
^3 For purposes of project management over the years, the studies 

were organized into four categories: learner* strategies, training 
strategies, validation strategies , and computer systems strategies. 
In learner^oriented research, investigators sought to find the 
students' cognitive and personality processes reflective of computer- 
based training. Therefore, studies included rule learning behaviors, 
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'. . ^ 2 
behavioral objective learning, memory, subjective organization, 
anxiety,* and curiosity, t Those involved in training strategies 
..studied effectiveness of computer-managed instruction, the possi- 
bilities of adaptive testing ahd modeling > and the vises of Simula- 
tion and information retrieval systems ♦ The focus was on the. design, 
implementation, and.>»7£l^tion* of CMI components, The ( area of 
validation studies led to the evolution of a Naval Research Inter- 

disciplinary Team, an important ideational contribution in its own 

• * 
right. Computer strategies produced, over the years, insight into 

interactive systems , instructional management systems , and data 

analysis systems. . 

Brought -together, the strategies, the studies, and the 

results lead to the repetition here, and throughout this^ report, 

of those themes which yielded the most profitable outcomes. 

1. , A model of computer -man aged instruction yields a sufficient, 
integrated, and cost-effective approach to training. 

2. The individual difference variables reflective of the 
behavioral processes of memory, anxiety, curiosity, and rule 
application are 'critically important to a *CMI .adaptive training 
model. Further researches requested to extend the set of avail- 
able indices reflecting training in learning behavioral objectives, 
graphics, etc. ' J % 

3. The selection and integration of media components and the 
requirements for' optimal resource allocation of training elements 

* (instructors, simulators, peer instructors, etc.) have received 
limited research attention and represent promising areas of the 
future*. * r - r * 
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■ - 4. The requirement to build substantive l?aison models between 

the Navy and university researchers is essential if the design char- 

, , , . » 

, acteristxcs and requirements , of training on the one hand and the 

theore-tical concepts on the other are to merge in productive fashion. 

T{ie PSty Interdisciplinary^^^^Y^l^search Team \*as an exemplary model 

of this liaison process. 

5. In reference to computer strategies, the creation of manage- 
\ment or central programs was highly profitable and the need to 

create training-oriented management systems is of the highest impor- 
tance. 

6. The developmental techniques to implement CMI were originated 
and should be extended especially for semantic models for natural 
language interaction and online manageTtfent review of adaptive 
training procedures . 
l.U Interdisciplinary Efforts 



A 



The DOD Themis Program was init/iated to facilitate the growth 
of university interdisciplinary efforts. From its inception, the 
FSU team reflected this interdisciplinary DO'D them$ and the require- 
ments for it in an area like computer-based training. While no 

formal evaluation was attempted, each researcher did comment- on the 

, ' ■ ' - ./ • 

intellectual benefits Qf the FSU interdisciplinary project meetings. 

V * 

Moreover, the benefit/ of collegial critiques, assistance, and joint 
studies was highly productive. From this person-to-person view, 
the interdisciplinary team w^s outstanding. 

As viewed at the university level, the interdisciplinary 
effort tended to be a failure. First, the ^xi^stence of inter- 
disciplinary institutes and groups at FSU diminished .markedly over 
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the last five years. Secondly, interdisciplinary 'research work 
is commonly treated as an overload activity to be pursued after 
the departmental duties are Completed. Finally, the recent fiscal 
stresses on universities have led to a traditional retrenchment 
that combats all interdiscijplijlary innovation. All in all, the 
university climate today is quite negative towards interdisciplinary 
efforts . These observations would probably hold for the majority 
of African universities and are. not unique to FSU. 
2.0 Learner-Oriented Research 

As the prior section, of this report indicates, the primary 

.focus of investigation for the learner strategy research was de- 

■ 1 <$ 

i 1 

voted to studying the internal cognitive and personality processes 
of the learner while in computer-based learning tasks. Studies of 
ru^Le learning behavior, anxiety and curiosity, and mem^ry^ lead to 
positive indications that can be predictors within an ^adaptive CMI 
system. Results of research into the role 'of behaviorklr objectives 
in learning, and into subjective organization have led investigators 
to view these as less rewarding areas .° 
2-.1 Rul^ Learning gehaviots 

PrWmfely the most common type/of learning undertaken by 
students is the acquisition of, principles or rule?. By learning 
rules in a specific situation as one of a class of situations, 
the rulfe-governed behavior of the learner permits more effective 
learning strategies that generalize to more than one situation. 

In this discussion of rule learning behavior, it will be 
helpful to consider three interrelated concepts. These ^re the 

" • 'A 

rule s statement, the rule, and rui'e-governed behavior,, A rule 
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statement describes the pro*.. :dure to be followed in. performing a 
specific operation on inputs from a specified clasfe of inputs to 

r _ ■ 

produce a specific output from' a class of- outputs • The rule? i^ 
considered to be the procedure or operation described by the* rule 
statement. Rule-governed behavior is that behavior that' would 
result from a student correctly applying the rule, "the ambiguous . 
term "rule learning" may ijow be divided into two appropriately ■ 
descriptive terms: (a) learning the rule statement which refers" 
to learning to verbalize the rule statement, and (b) acquisition 
of rule-governed behavior which refers to the correct application 
of the rule, - ' , , , 

At FSU, a study was implemented to investigate prior findings 
that presented rule statements reduced the humber of examples needed, 
and f total time required to meet a' prespecified criterion. Further, 
presentation of rule statements increased ' performance on a transfer 
task and reduced the requirement of, reasoning ability in learning 
the t^sk. The presentation of objectives also reduced thfe number of 
examples and reduced the requirement for . reasoning ability. . The « ■ 
stuc^ showed jthat pf esantation of rule statements reduced the num- 
ber of examples reqtaire'd to meet criterion performance^ increased 
posttest performance, reduced tot^L time required to meet criterion 
performance, increased retention performance, and reduced' \the' level 
of state anxiety within the learning task* \ 

A further study in this area found that the presentation of 
9 - \ ' ' 

objectives increased- the total amount of time spent studying a 

rule learning • task. The presentation of sample test items within 

the learning task reduced the requirement for memory, but neither 

..." ' «v ' • * 
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objectives nor sample test items had an effect on post or reten- 
tion performance. • 

Implications from these studies for instructional nathod 
in acquisition of ryle application. -skills appear to be v threefold: 

1. Instruction should present general instructional objectives 
to inform the student as to what is expected of hiiji but not for 
specif ic learning. 

2. Presentation of rule statements can prevent the "discovery" 
of an incorrect rule. 

3. Presentation of sample test items gives the student a chance 

for practice and immediate feedback. 

. . . 

Further studies' at FSU revealed that availability of prior r - 
examples reduced problem response latency afnd redug/ad the require- 

• V 

ment ton^ reasoning ability. A present investigation appears to show 
that memorization of work statements reduces both the number o£-~^*~ 
examples and total time to reach criterion performance on complex 
rule application tasks. However, no memorization effect has been 
found on the posttest or retention test of rule applidation skills. 
Implications for instructional development are (a) the requirement 
of memorization of rule statements to facilitate the correct choice 
of a rule when more than one is being learned, and to aid in recog- 
nition of operational symbols if they are used is of marginal bene- 
fit; (b) the use of several examples to show the statement instances 

of the correct application of the rule is critical? and (c) the 

1 

use of review if more than three rules^ are being learned in one 
session to provide simulated testing i^ required.. 
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2.1.1 Future 

In addition to apparently useful generalizations given 
° for the development of instruction for acquisition of complex 
rule application skills, implications for fruitful research in 
this errea have grown out of tfye FSU investigations* It should 
be ascertained what ty e(s) of examples are most profitably pre- 
sented to the student: nonexamples , examples, or both* A 
second topic of research recommended is the investigation of de- 
sign of rule learning materials dependent upon the abilities of 
the learner; this would contribute to the adaptive model research. 

A final topic which appears to offer promise is the area of optimal 

t , 

number of examples for presentation before sample test items. All 

of these studies relating to number of examples would contribute 

* 

to a drill and practice adaptive model. 
2 . 2 Behavioral Objective Learning 

Only in the latter pai^t of the decade was research under- 
taken to examine empirically the claims made for objectives in 
instruction. Of the approximately 35 studies conducted, about 
half failed tc confirm the hypothesis that providing students with 
objectives leads tc* increased learning. Remaining studies showed 
facilitative effects. -The research on interactions between ob- 
jectives and type of learning provided few positive findings. Ifo 
general conclusions are evident in the research, which found a few 
significant interactions between objectives and learner character- 
istics. 

^ . Investigations at FSU found ^iia^pttjectives significantly 
increased study' time with no post^est difference between groups, 
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and that objectives partially reduced anxiety without affecting 
performance. The hypothesis that objectives add structure to 
learning as is done by advance organizers wa^s not confirmed ; A 
* further study found that objectives focused learning c^n relevant 
materials and depressed incidental learning. 

To generalize from studies^overall , it seems that objec- 
tives have only a peripheral effect on learning, and this effect 
does not appear to be as strong or as pervasive as originally 
assumed. 
2.2.1 Future 

It is anticipated that research "on behavioral objectives 
will continue the trend away from exploration of .their effect on 

. I 

learning. The frend will be toward exploration of interactions 
with task and learner characteristics. The exploration of hypoth- 
esize^ cognitive functions which may be fulfilled by objectives 
in the learning situation is anticipated to grow', but to have 
limited positive potential. 
2.3 Memory Research 

An important capacity of memory is the ability to discrimi- 
nate. This is an idea which has been reinforced by the research 
which was conducted on direct forgetting. Important to the retriev- 
al of some previously stored event is that the individual be able 
to discriminate it from other stored information. .Voluntary for- 
getting appears to be largely a matter of sharpening the capacity 
to discriminate between material which is to be remembered and that 
which is not. 

This way of looking at memory raises the question of the 
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nature of information storage, if the ability to discriminate 
among memories is important, then perhaps encoding is in terms of 
dimensions which offer a basis for discriminating among memories, 
and more ^specif ically , in terms of the values of such dimensions. 
Just what these dimensions are is unknown. The point is simply 
the proposition that events are encoded according to abstracted 
features. 

Appther study investigated short-term memory (STM) pre- 
requisites to development of more complex mental processes. Of 
special interest were the possible differences between white 
vantaged children and* black, disadvantaged children. Diffe 
would seem to indicate that memory characteristics' are 
samqt for these two groups. 

The findings were also compared with prediction' of three 
of the more characteristic information processing models for STM: 
Sperling's Visual S^M Model, the Feigenbaum-Simon Computer Simu- 
lation Model, and the Atkinson-Schif f rin Model., It -was' found that 
some of the results could be incorporated into the structure of 
all models, but that no one model could account for all the findings. 
An alternative model was developed which included an IQ parameter, 
a more cipmplex rehearsal process, and an operational representation' 
of organizing strategies. ^ 

Conclusions for both the directed forgetting and STM J 
research are that they both indicate possible individual differ- 
ences which could be useful in adaptive instruction. It *is nec- / 
essary to continue research to fully operationalize appropriate 
memory indices. / 
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3 . 1 Future 



In the near future of memory research, the computer will 
be used to construct theories and to simulate memory performance 
rather than to assist in the conduct of experiments . This will 
be particularly true in the a^reas of semantic and long-term 
memory. These theories are particularly complex and it Will be 
especially useful to simulate performance. By contrast, exper- 
iments to test such theories may be simple and fairly easy to . 
conduct with relatively unsophisticated equipment. 
2 . 4 Subjective Organization 

The purpose of the research on subjective organization 
(SO) was to determine if SO is a meaningful individual difference 
variable in the* learning *df verbal discourse. Furthermore, the 
relationship between SO and other aptitude variables^ thought to 
be important determiners of learning from prose materials was 

examined. In order to accomplish this, a new measure of subjec- 

j 

tive organization had t$r be developed because previous measures 
had certain limitations which were thought to mitigate against 
finding a meaningful relationship between memory organization and 

s 

^recall. Thus, as an outcome of this research a new measure has 
been examined and partially validated in relation to other SO 

measures and other individual dif f ejrence 'variables * 

* * 

J 

' The basa^p research paradigm was one in whifch SO and other 
measures were administered to subjects and then those subjects 
were asked to learn paragraphs which were organized in various 
ways. The results indicated that high organizers are highly in- 
fluenced by the external structure of the learning^materials , and 
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they tended to mirror that organization in their recall* ' High 
organizers, however, were* able to . reorganize the materials , but 
that^Was not to their advmt^e because of the time constraints 
inj/olved. That is, high organizers actively sought alternative 
relationships among the sentences, thereby reducing the amount 
of time spent memorizing 'them. 

Four generalizations can be .made on the basis of the 
studies which were conducted at FSU on subjective organization: 

l r 0 The ability to reorganize material can be a limiting 
factor to a learner if sufficient learning 0 time is not provided. 

2. Low organ4zers need to have learning materials structured 

^ ■ . t ■ 

to ->a. higher degree in order for them to learn "effectively* 

z\ * • ' * 

3. ' High organizers, perform equally on materials of high and 
low structure. f 

4. Our current measures of SO are too limited and lack ; 
sufficient validity. , ■ ■ 

2.4.1 Future * 

The conclusions of the investigations in this area indicate 
that future research should consider the following: 

1. A practical, reliable, and vali{j/!b«strument for measuring 
SO must be developed. -* In this ^regard, the potential'' use of a 
computer for realtime presentation 'and scoring is promising* 

2. Explorations^ on more complex instructional tasks must be 
undertaken on the- finding that low organizer^ need highly struc- 
tured materials whereas high organizers j^e^form the same on both 

\ 

high and low structured materials. 
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Continuation of research in these areas should provide 
instructional designers with a set of findings which would guide 
the structuring ,of textual materials sucfr that the learning of 
all students will be maximized. . \ 

2 • 5 Anxiety and Curiosity Research 

The FSU investigation of anxiety and curiosity behaviors 
of -a student while being trained was primarily focused on the use 
of these personality indices as predictors of optima]; instructional 
treatments; While state/trait theory initially guided our efforts, 
the subsequent inconsistencies between replicatable findings and v * 
the theory indicated its limited role in a/draining context. The 
theoretical study of curiosity did provide us with a new framework 
which appears to resolve many of these prior inexplicable com- 

. . * ; . 

plexities. c 

The development of state/trait anxiety theopy, based on 
drive^theoryv allowed a number of derivations for prediction of 
performance relevant to computer-assisted instruction. At the 
same tirrie CAI seemed an excellent vehicle for testing (the theory. 
The resulc^were not encouraging for confirming a driveVtheory 



interpretationNof state/trait anxiety. However/ anxiety^ was shown 
to be an indicator of CAI performance and the thrust of research 
continued in this area.^ Studies included anxiety effects on memory 
support, test taking, and interactions withnr^a^onse mode, subject 
matter familiarity, and learning timei ■ 

The important results indicated a consistent inverse re- 
lationship between state anxiety and learning, differential .effects 
of trait anxiety especially as it revealed learner latencies , the 



complex "role of memory aids and response modes f as facilitators 
under some conditions, and the complex functional relationship 
of anxiety, curiosity, and latencies. This latter finding could 
be a promisihg future framework for studying latencies within 

training* ' ' 5 " ' i\ 

In addition, the anxiety theory and research directly > 
initialed and influenced new work on state/trait episteraic Quri- 
osity. Curiosity provides an additional indicator of performance 
within instruction and specifically CMI. The concurrent interests 
in adaptive, instruction models resulted in postulating the value 
of thes^ personality variables as useful predictors in decision 
models fox selecting alternative individualized treatments-. One 
study, utilizing regression techniques, especially found state 
anxiety, to be a useful predictor for selecting remedial instruction 
2. \5.'l Future ■ " - 

V^jFor'the future, the investigators anticipate that the" 
effort within adaptive models and the theory, of anxiety-curiosity 
within u training will be highly^ f ruitful . The use of anxiety- 

> \ ' ....... ' ■ - i * • - 

curiosity concepts to study response latencies may help explain 
the highly variably results thus far found. 
2.6 Graphics" Research 

.In \ order to employ the research capabilities of • computer 
presentation of graphics in instruction', an initial stud^at FSU 
provided the opportunity for devising techniques to code graphics 
onto the' cathode ray tube screen. This general model .of graphic 
encoding was\ then used to develpp studies of selection and se- 
quencing of graphics for realistic training materials. Further 
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attempts to learn the rules of graphic learning revealed that 
tire search for such rules is difficult, and can ever: be con- 
sidered unprofitable. 

Furthering the graphics investigations, FSU pursued re- 
search in mapping sentences onto graphics. Studying this in- \ 
structional technique provided data which were complex. lore 
appeared to be going on in this instructional situation than 
couldtbe easily interpreted by learning theory. 

Used as a v teaching aid, computer graphics proved helpful 
in instructing engineering dynamics students in the traditional 
stumbling block of Euler angles.. Graphics assisted in visuali- 
zation of the angles, and subjects reported positive attitudes 
toward the instruction. 
2.6.1 Future 

Generally, given current constraints, the "pearch for rules 
in learning from graphics appears to be unsuitable for study... As 

s 

an aid in problem solving, graphics apparently lend assistance to 
lelrning and the experience can be positively viewed by learners. 
3 . 0 Training Strategies 

A major investigation of computer-managed instruction was 

v » : 

j . * . # 

conducted as a priority in training strategies, to study tfraining 
effectiveness, and cost benefit outcomes. Further training stra- 
tegies in the areas of adaptive testing, adaptive systems, simu- 
lations, and ' information retrieval system.^ were, also researched 
to various degrees. The research, generalizations, and future 
trends are described in the following sections. 



3 . 1 Computer-Manag ed Instruction 

Throughout the duration of the contract there was a con- 
current set of investigations in computer-assisted instruction 
(CAI) and computer-managed instruction (CMI) . The primary pur- 
pose of the investigations of CMI was to determine its training 

effectiveness and associated cost benefit outcome in comparison 

t 

with CAI and other more convientional means / of instruction. As' 
pursued at FSU, computer-managed instruction involves the followin 

1. diagnostic, assessment and the assignment of individualized 
learning prescriptions , 

2. the use of CAI for practice and remediational purposes, 

3. .the use 'of simulation for role and decision-making 
training purposes, 

4. the use of the computer for ease and objectivity of 
curriculum development , and 

5. the development of a record system so that the individ- 
ualized training process can be effectively monitored and 
managed. . * 

Within this CMI conceptual context, a number of studies s 
were pursued. All of these studies indicated that CMI at the' 
collegiate level is highly feasible, cost effective, and provides 
for learning results similar to CAI. However, due to the mastery 
level learning approach utilized in the instructional, Materials , 
the relationship oti individual difference variables to learning 
rate or performance was more limited. Where extensive media and 
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recitation sections were used, the effects of individual differ- 
ence variables seemed to be more pronounced. Finally, learning^ 
attitude toward the instructional materials was quite positive 

and could be manipulated by the form of training, ^ 

• ✓ * 

In turn, investigations* of CAI indicated that it is use- 
ful in a number of technical training areas. It proved espe- 
cially useful for dynamic graphics such as found in engineering* 

. - i 

dynamics. However, while CAI was shown to be visible in areas 
like chemistry, the results did not tend to ex<^*d those fox 
in CMI. 



v 

It is important to note the fact that the development pro- / 
ce'ss for CMI, while not q.uite as demanding as that of CAI, stall 
\was considerable. Dependency on a sound training model, use of 
formative evaluation, and effective monitoring pf students in an 
individualized mode seem to be the critical factors in the design 
and implementation of CMI. * , a 

From this, consequently, the following research' generaliza- 
tions can be derived: 

1. Terminal-oriented computer-managed instruction has shown 

to be more effective than conventional instruction and less costly 
than computer-assisted instruction. 

2. The most significant gains in the quality of instruction 
have not necessarily been due to the use of computers, but have 
been through the implementation of systematic approaches ,to the 
training process required for application of the computer. 



3o Although the computer provides the instructional developer 
with more information about the instructional 'process than has 
been available, the revision process remains the least well under- 
stood fand utilized component of the systems approach; however, 
the provision for systematic, reliable data encourages attention 
to this problem. 

4. Interdisciplinary collegiate development teams will not 
necessarily produce better computerized instructional materials 
than those produced by conceptually integrated teams. 
3.1.1 Future , 

C During the time period since the initiation of the Themis 

Project, educators have come to understand that -in or N der to imple- 
ment,, individualized instruction fdr the learner, these is a need 

0 J 

to provide similar training £c?r the trainer. .This new foxm of 
teacher training has ^een labeled "competency-based" or "performance 
based" training. The implementation of such training program? 
has more clearly identified the needs which such programs have for 
a computer. These computer needs, in general order of priority, 
are listed below: 

1. > Record keeping,- of computerized and noncomputerized assess- 
ments of student performance for monitoring purposes 

✓ / ' *• 

2. Scheduling / 

* " / ■ 

3. Assessment a£ "computer-testabl^" skills 

4. Adaptive remediation, including CAI and drill and practice 
exercises. '* / * 

If this list is an accurate reflection of tfte needs of 
such programs, then it would appear that the role \f the computer 
O 9 ' 
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must be first as a general management system and only secondarily 
as an assessment ; and "instructional tool. - x '" 

3JL 2 Adaptive T esting ' 

\, Th£ initial undertaking with adaptive testing concerned 
itself with the automation of intellectual and personality measures 
This effort successfully resulted in -the automation of an indi- 
vidualized intelligence te£<t (Slosson Intelligence Test) with 
satisfactory reliability and validity coefficients. Moreover, a 
complex personality test, namely, the MMPI, was also satis fac- - - 
torily implemented* These studies indicated that response latency 
contributed to the score information and concurrent validity when 
measured against a conventional administration. This line of re- 
search eventuated in a systematic review concerning the various 
uses of automated testing. - 

In turn, the development of sequential testing and tailored 
testing continues to be investigated. Utilizing simulation tech- 

a' 

niques, sequential testing can be shown to be a profitable approach 
The general goal of sequential testing is to present the smallest 
number of test items necessary to accurately classify an examinee 
into two or more exclusive groups, or iin conceptual terms , to im- 
prove reliability and validity of or the decision process within 
CAI testing. Previous computers-bashed sequential testing models 
have been based upon right-wrong measures of performance . . The FSU 
Sequential testing effo>£ d iffers from previous models in that 
both univariate and multivariate performance measures are utilized 
The multivariate prediction model fcs expected to increase classifi- 
cation accuracy beyond that possible using binary-coded item re- 
««j«se data .alone. More importantly , the, use of tailored testing 

ERIC . , 



is currently being investigated as an approach for the . administra- 
te " ■ ' ' ' t \ 
tion of attitude items in ;a training situation. It is conjectured 

that the number of test items can be minimized and the assessment 

of learning satisfaction can be improved ifa terms of its accuracy. 

These* studies of sequential ^eating led to the development 

of adaptive 'reading cc&mprehensibn te^ts. , The adaptive reading * 
* • •* * > * 

tests .provided" for a more dynamic individualized search of the 
student's current comprehension level , especially under paced con- 
ditions. Th^sc^tionale for using paced conditions is based on. 
learnin^time* savings being pace contingent. 

3. 2vl Future 4 .' v ■ , * 

■ * 

In the ^future adaptive testing willJtiost likely focus oh 
two aspects . 'First, an ^investigation of inccprrect as well as 
correct alternatives v^a Bock's multiple category model Sibws 
great promise.. This approach shoulcLallow for adaptive selection 
of items hy item alternative characteristics and the assignment 



of sequentially derived confidence bands about the student's per- 
formance level/. Given tAe improved information from this approach, 
it is then conjectured- that a sampling" approach to .sequential 
testing c£n more effectively provide for the early classification 
of a subject especially if complex confidence band techniques are 
utilized. Thus r the' f uture tends to hold a continued seeking for 
more efficient testing procedures as well as one^ which^ wiJ^l have . 

psychometric properties appropriate for the individual and not just 

■:. . . t 13 & ^ ■ a v 

the grotfp.' ' / ■' . 

3 . 3 Ad ap t i ve v Mode ling; * . , " " t 

•4 p f * I ' 

A major theme throughout the ONR-sponsored, research -hati* 
been that** instiuctipn should not be fixed for all students either $■ 
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in treatments or time. The alternative to fixed instruction is* 
adaptive- instruction which attempts to match an individual's 
unique characteristics with optijnum treatment dimensions. Models 

for selecting appropriate alternative treatments consist of con- 

* ■■ 

, \ 

tingent, mediated, and optimizing algorithms designed to achieve 

t 

the objectives. 

The initial studies of adaptive models at FSU investigated 
learning and personality variables which miglvt prove useful' in 
predicting and assigning remedial instruction, lessori\ length , and 
other instructional variables. Multiple 'linear regression methods- 
were developed and used in an early modeling attempt with predictbr 
variables consisting primarily of performance variables. Person- 
ality -variables were added, in a phased approach to determine u^e^ 

ful predictors, which resulted in outstanding performance levels 

{ 

for the adaptive group. A major study which manipulated the assign- 
ment of remedial instruction was undertaken. The order of outcomes 

** * 

in terms of superiority were the adaptive model "strategy ] remedia- 
tion for all strategy, a learner choice strategy, and a no- 
remediation strategy. 1 

tjiese results and models were then considered for employment 
in reading comprehension. . More recently, models 'based on the 
experience gained' in the ONR-sponsored research we redeveloped 
for the Air Force Advanced Instructional System. In addition, 
OttR adaptive modeling has continued with an investigation of drill- 
and-practice in terms of lessop structure composition. Further, 
the study of the utility of measures which relate information 
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processing to reinforcement contingencies for adaptive instruc- 

tion has resulted in a model which takes into account both memory 

and motivational processes and procedures . 

3.3.1 Future 

It is anticipated that investigation of the monitoring 
and n?.naging aspects of both training resources and of the indi- 
vidual will provide payoff in the future. This will involve 
managing such alternatives as the problem set size, the sequences 
among sets, the assignments of media, the assignment of instructor 
arid peer time, and the use of incentives . 
3.. 4 Simulations 
3.4.1 System Simulation 

One of the major methods of using simulation in education, 
as well as other spheres, is the projection and analysis of systems. 
By simulating a prototype computerized management- system intended 
as a sub^oi(^^ient of the FSU elementary teacher training model, 
the^AI Center -examined a number of problems in data acquisition 
and instructional systems. The following generalizations resulted 
from the system analysis: * (a) a computer-managed system is tech- 
nically feasible for use with an individualized teacher training 
program, and (b) computer processing for trainee scheduling and 
testing data, as well as program management, is necessary,. 

Simulating a teacher training system using information 
from a behavioral simulation, FSU found advantages in experimen- 
tation capabilities, and resulting prediction^ about the teacher 
training program. Further, advantages were found in the utilization 
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of regression techniques in combination with A Programming 
Language (APL) for designing , constructing, and utilizing a 
system simulation. C 
3.4.2 Behav i oral Simulation ^ 

A behavioral simulation was used at FSfr ^o t rain prospec- 
tive teachers by providing an environment facilitating transfer to 
the classroom. Analysis for the behavioral simulation proceeded 
through assessment of entry behaviors, collection of data on 

trainee performance during instruction, collection of data on 

) • 
criterion variables, and analysis and , evaluation of the required 

training process in terms of the dimensions of simulation theory 

to estimate the nature and validity of the behavioral simulation. 

Behavioral simulation appeared to permit change in participants 1 

behavior patterns. 

3.4/3 I nstructional -Simulation 

A further "major function of simulation is instruction. 
Much of the debate in this area continues to be over the poten- 
tial for simulation to affect attitudinal and/or-cognitive growth. 
One game , or simulation, developed at FSU was designed to allow 
the player to relate his knowledge of .science to situations in 
elementary classroom instruction. Use of the game by graduate 
students indicated that a learning environment of this nature 
can foster growth in both cognitive and affective areas. 

One of the primary S^vantages seen in instructional simu- 
lation is the ability to reduce cost (e.g., by reduction of ox- 
pensive laboratory equipment) and a second is ^experimentation 
without danger (e.g., investigating dangerous chemical combinatipns 
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without use of the chemicals) . Investigation into simulating 
laboratory experiences at FSU showed that simulation, as opposed 
to traditional laboratory methods, was equally effective as 
evaluated by posttest performance and total instructional tine*' 

3.4.4 /Stat istical Simulation 

As is suggested in the Computer Systems section of this 
report, A Programming Language (APL) proyes to be an outstanding 
computer language for the simulating of statistical analyses, duo 
to its mathematical capabilities. It is possible for a student 
to explore the meaning of various statistical concepts through 
sample problems and exercises, which he works as often as ho 
wishes. He can be guided through the hypothesis testing procedure 
by inspecting data, successively making appropriate inferential 
decisions as further hypothesis testing information is given him. 
This type of APL simulation continues to expand as its utility 
and appeal are exposed to wider audiences at FSU . 

3.4.5 Simulation Development 

Investigators of simulation generally predict a more wide- 
- 
spread use of the technique, both in education and in other spheres, 

for research and analysis as well as instruction and traininq. Ad- 
vances in the development of such simulations, however, are meager. 
The major problems affecting progress are seen in evaluation: 

1. Many authors feel that their simulations produce affective 
^rather than cognitive changes, and ajre unsympathetic to stating 
behavioral objectives. Since an equal number of authors and 
users request behavioral objectives for simulation, this problem 
may not be resolved until statements cf affective objectives are 

eric 



satisfactorily developed, and can be evaluated. 

2. Evaluation of instruction by simulation is ^correspondingly 
difficult. A search for satisfactory new methods of evaluation 
may be a future activity, or a satisfactory relatinq of simulation 
evaluation to existing techniques. 

3. The validation of the simulation model itself remains the 
most difficult part of simulation development. While mathematical 

^.simulations appear to have potential to solve this problem through 
output comparisons, simulations of human behavior, for ^xam^ie, 
provide a requirement for establishment of other model validation 
procedures. * j| 

3 . 5 Inf onaation Retrieval 
3.5.1 IR for Inquiry 

Information retrieval (IR) systems, like simulations, can 
be developed and used for research and/or instruction. FSU re- 
search on a CAI-based information retrieval system containing 5312 
social science generalizations was directed toward examination of 
human inquiry behavior and appraisal of affective factors within 
the task* Both .the feasibility of the approach to improving inquiry 
behavior and the positive nature of learner reaction were estab- 
^ Tished in the IR application* 
3. 5,* 2 IR Instructional Tasks 

In the library science area, recognition of the i'uturc 
growth of automated IR systems led to provision of computer-based 
IR tasks for advanced students* Both instructional and research 
goals were set for the use of the online searched coordinate in- 
.dex, which referenced 1856 documents orior to its transfer to the 
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IBM 6400. As a teaching aid in graduate courses, the IR system 
served several instructional purposes: (a) a demonstration of an 
cr.-line search was available to students, (b)'the system provided 
lessons in preparing and searching coordinate indexes, (c) the 
system permitted students to develop skills in the s use of com- 
puterized IR, and (d) students were able to perform test searches 
of the index with reference questions. 

For research purposes, all student uses provided data which 
were collected by the computer system and stored for use in i 
evaluation and revision. Data collected on student attitudes were 
positive, as were professional observations^ This system was one 
of -the first implemented in library schools, 
3.5.3 Future 

Since good inquiry skills are universally useful, the po- 
tential of IR systems in enhancing these skills is an area of 
interest in which future research is ^commended. It is suggested 
that the new techniques of observation $and evaluation gained in 
participation in these developmental efforts become part Of the 
individual's professional research, development, and teaching 
approach, thereby enriching members of such groups. 

The development and use of information retrieval systems 
in library science is an example of uses that can be' macte of 
specialized IR systems in a number of fields. Since the. number 
of fields which will add computers and IR svs terns to their require- 
ments continues to grow, the development of thi.s type of instruct ion 
should bo present in concomitant instructional areas. 

ERjfc 
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4.0 Validation Strategies 

During the Themis Projebt, every effort was made to inter- 
\ relate with ongoing Navy training, its associated problems, and 
^its need for research. Contact was maintained at a 'professional 
level with the 'Navy -Research' ' Laboratories at San Diego, Memphis, 
and Orlando. As an integral part of Themis, a team of FSU profes- 
sors who were also active in ..the Navy Reserve was formed into a 
functioned, unit. This-" grotfp represented a bridging process between 
the various research studies formed. here at FSU and their applica- 
tion within t/he Navy. Secondly, experience developed under zns 
ONR Thenks Contract allowed perfprmance of associated Department of 
Defense activities* The most significant of these was the o design 
and specification for the Advanced Instructional System at Lowry_ 
Air Force Base. \j * , 

4 . 1 Navy Research Interdisciplinary Team . x 

The team of FSU professorial investigator's, who also had 
active relationships in the local Naval Reserve "pursued a number 
of objectives. First ari& -foremost , the. group applied the concepts 
and /procedures of computer-managed instruction (CMI) to local 
training of the Seaman 1 classification. This CMI implementation 
study was quite successful in that there v^as both a significant 
increase in performance level as well as a significant savings in 
training time. The results werCteni^red ' by the\small number of 
participants and areola th^ process of being replicated at such* 
bases as the Air-Naval Station at Memphis, Tennessee. 
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Secondly, the group pursued broad systems concepts for the- 
planning and management of Naval training. This led them to * 
attend many conferences and provide consultative help to the Naval 
Reserve sysJ^Jm itself. In many cases the team members performed 
active duty at Omaha* which led to significant plans whereby more 
effective reserve drills couldf take place. p 

Lastly, the team providecKieadership "for the Navy concern- 
ing university and community relationships. As university models, 
they effectively contribute to and are utilized within Naval 
Reserve operations. Secondly, in regard to community -involvement/ 
fhey advanced the Navy concerning such special training problems 
as race relations, drugs, and either associated behavioral matters. * 

A V 

Perhaps more directly related to computer activity, they assisted 
in the Resign o£ information management systems and associated 
adaptive training and testing systems* «■ 

In regard, to some constraints, the usual problems of communi- 
cation and lack of goal understanding handicapped the group at many 
turns. Given a fair degree of isolation in Tallahassee, Florida, 
the. group maintained a vigorous information exchange which undoubt- ; 
edly led to the successes that it achieved.' 
4.4.1 Future 

For the future, this university -Navy liaison should be 
pursued with more vigor. ^ The benefits of mutual problem solving 
are sufficient to make it an essential part of any large training 
R and u effort. The two-way contmunicational and ^educative process 
.has the highest potential. 
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4 . \><A) CoTftputor^k'j Idt od Actiy i ties 

As-an indirect consequence of Thomis, lhr projoct porsnnnn] 

was also able to serve the Department of Defense on other train! nq 

projects. The most significant one of these was the AF Advanced 

Instructional System (AIS) . »FSU could not have de^ifjnerl and spec-. 

ified the AIS without the prio'r work of the Lowry Human Resource. 

.Laboratory group or its own critical experience and data from 

Themis. A descriptiorkof the AIS project follows . ./ 

■ r % - ■ ' 

The Advanced Instructional System (AIS) is a development . » 

within the Air Force Human Resource^ Laboratory (AFHRL)' to imple- 
merit the latest demonstrated state-of-the-art training techniques, 
media usage, management procedures, and computer technology to 
Air Force Technical Training. The CAI Center was awarded a con- 
tract by AFHRI> to develop functional design specifications for the 
individual multimedia computer-based training .system which would 
provide significant cost-effective improvements in the operation 
of technical training courses at Lowry Ai v Force Base, Colorado. 

In addition to the goal of providing individualized train- 
ing, AIS will focus ♦ on the managerial processes whiofc ^can be en- 
hanced by the computer, cost-effective multimedia approaches which 

... . - > 

may provide tune-savings, modular implementation which will provide 
both flexibility during development and revision Of learning mate- 
• rials and additional cost savings during expansion throughout the 
Air Force* , \ ' 

v. AIS will be implemented within three technical training 
courses with a total enrollment of -over 2000 students. The courses 
t>£ Inventory Management (inventory and supply) , Precision Measuring 
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Equipment (precision electronics measurement and calibration) , 
and Weapons Mechanic (tactical weapons loading) represent a 
broad range of technical training requirements* 

The AIS consists /of seven subsystems which reflect the 
scope and complexity of the effort: (a) instructional materials, 
(b) r instructional strategies, (c) media ^hardware and software, 
(d) management components, (e) computer hardware, (f) personnel 
and training requirements, and (g) related requirements, these 
subsystems are designed to provide for all aspects of the, instruc- 
tional process from materials development and evaluation through 
student use and management to review and revision. Operational ^ 
considerations, including 'computer systems design and maintenance , 
CAI languages, and the selection and traiping of AIS personnel 
are, als6 considered ih the design. 

5.0 Computer Systems Strategics 

There were three main areas of design and impitementation of 

* . ■ v 

computer systems for instruction, namely, (a) interactive pro- 
gramming languages, (to) management schemes, and (c) data analysis 
systems. The softwaz'e or language developments employed a common 
strategy, to maximize the ease of use while devoting a minimum of 
developmental effort. The topics within this theme will be organ- 
ized around the above* three topics . 

5 . 1 Interactive Systems ' ^ 

5.1.1 Coursewriter Functions y 

Of the institutions utilizing the IBM 1500 system, FSU 
probably developed the least number of additions to Coursewiri ter 
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capabilities in- the form of functions. This is^due to the frame- 
oriented Cou'rsewriter language and the fact that the already 
available functions were sufficient within limits. 

The frame or PI orientation of' Coursewriter is important 
to recognize because of its tendency to direct the type of in- 
structional material and strategies for both learner and researcher/ 
developer. A major ^lesson learned from vise of this language was 
the need for non-frame oriented, algorithmic style languages when 
expansion of research and development is the goal. 
5 . 1 . 2 A Programming Language (APL) 

The development of the programming language APL (A Pro- 
gramming Language) on the IBM 1500 CAI system was a project 
oriented toward determining the capabilities of this type of lan- 
guage for 4 meeting the diverse requirements of computing in educa- 
tion. The elegant and powerful dat§ structures and commands of 
APL appear suitable to the areas of instruction, data processing, 
and computer science. 
5.1.2.1 A PL Generality ^ 

A measure of any programming language is its need for ex- 
tensions. In the- instance of APL, extensions in the form of func- 

T - * 

tions were developed -in the areas of programming *aids , statistical 
analyses, character handling, numerical manipulation, and graphic 
control. The fact that such APL functions can be created and so 
-extend the language is to its credit; however., the*fact that exten- 
sions were needed is an indicator of educational requirements for 
more than one computer language. 
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5.1:2.2 APL Instructional Programming ^ 

... * 

APL 1 s usefulness for statistical analyses and simulation 

activities has been documented by FSU, and demonstrated 'through 

certain graduate -level courses in the College of Education in 

which the majority of st'udepts chose to learn APL as a programming 

language for educational purposes. Beyond the use of APL for 

instructional simulations, however, there appears to be reason- 

to 

able doubt as to its feasibility as a general CAI language. In 
the experiences at FSU, little instructional programming with APL 
was performed other than simulation packages. 
5.1.3 Natural Language Processing 

One of the initial dreams for research in CAI was the 
possibility of nearly natural communication between student and 
computer for the purpose of instruction, Over the past few years 
this research has become actuality based on the concept of seman- 
tic processing and semantic nets as defined first by Quillian. 
Carbonelle terms most CAI systems "ad hoc, frame-oriented" systems, 
in that all stimuli presentations and responses must be fully 
planned ahead. With a natural language processing system and^ap- 
propriate semantic-oriented data bases, such preplanning is not v 
necessary, and, in fact, the flexibility of the system may be more 
powerful and realistic for learning. 
5.1.3.1 FSU Dialogue 

* The thrust of FSU natural language research has been not 
so much on the technical aspects ofs computing .that must be solved 
for such a system, but rather on the .pedagogical considerations 
for specifying and planning the desired type of instructional'. 
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dialogue. To accomplish this; a sy^&ieroatic technique was devel- 
oped wKich utilised the typicallyr stated. ,3 teps for instructional 
development , oriented tov/^ff'Tfiatural language instructional 
dialogues. More specifically, the analyst of instruction was 
based on determining a semantic network of information which the 
student was expected, to have learned by} the en$ of the dialogue. 
On the basis of this semantic network, a data base and questions 
to the studqnt can be specified as performance objectives and so 
programmed. While thi^ particular type of system is limited in 
its power for developing semantic networks , particularly for syn- 
tactic analysis of student input, the project was nevertheless a 
success as a prototype development. A complete dialogue was 
developed for a humanities coujrse, and students using the dialogue 

o 

x in this course did generally acquire the semantic content as 
specified in the systematic approach to the dialogue development. 
A more extensive research program would be required to fully re- 
search and evaluate the techniques used. • . 
5.1.4 Remote Comraunications 

of the major problems experienced with the IBM 1500 
instructional system was the lack of a teleprocessing capability. 
The requirement for teleprocessing became apparent as projects 
developed ffcr transporting CAI and CMI capabilities to rural ar^as 
in Nortfh Fffiprida' and to government laboratories. The cost for.v 
initial CAI capability was obviously beyond 'the financial reach of 
these institutions, and the cost-effective manner Of providing the 
capability was through the .FSU system. Thus** a remote teleprocessing 
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communications system was developed in house including both 
software and hardware design* PSU personnel worked with per- 



sonnel from Digital Equipment > Corporation in this project, as • . 

IB & :\ regarded/ these requirements; as impossible. The major lesson * * 
learned from this effort is. the necessity to consider teleprOcesr 
sing capabilities as an essential requirement for any computing * 
system vhich is- to be used in education* 
5=1.5 Computer Art 

Recently, the art community has discovered the ^computer 
in iips quest f orGigw jjiedia f or artistic involvements To provide 

support for this movement, in the belief that computer use in' 

» 

education .means more than prbgrainmed instruction, FSU convened a 
semioiaj: in computers in the arts* Art prototypes were produced. 
A notable^advantage which seemed salient was the development of 
interacm\fejbomputer art through which art observers could be- 
placed responsive and controlled art environment. This 

interactive dirtfen^ion of the mediumylends itself to the artist's 
creativity beyond other current medaa forms * . • ". 

•5.1.6 Future f 

/ It. is anticipated that further exploration to the area of 
interactive systems is very promising/ especially in terms of multi- 
media approaches. Present language capabilities will continue to ~J 
be expande'd, while more interactive dialogue and natural languages 
will be developed. > 
* 5 . 2. Instructional Management Systems 

. Instructional management systems consist of tiiB computing 
software angk hardware necessary for control f and monitoring o£*the 

o • > • * % 
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instructional process. Operationally, it is convenient to talk 
of the instructional data bas£ as the associated files within the 
data base as well as the manipulation cf those data for control 
and monitor functions. It is a prime tenet of computer systems 
analyses that the specification, design, and utility of instruc- 
tional systems data bases are tied to appropriate identification 
and integration to the educational functions. The focus of the 
FSU approach has Been development and dissemination of data base 
software appropriate to educational users in both service and 
research . • " 

5.2.1 AFL File System Package . . 

One of the prime limitations of the APL/1500 system, and 
indeed most APL systems, was the lack of a file establishment and 
manipulation capability. To overcome this limitation, the APL/ 
1500 File-Access Subroutine Package was developed. The package 
. permits of f line access to data generated through instructional and 
research applications of the system. , It was found that needs of 
users which must be , filled by the system are: (a) the capability ^ 
to access data for analysis by other computer systems, (b) the 
capability to access data for acquiring. hard copy in a quicker 
mode than is available_on^an IBM 1500 system, (c) the capability 
to build input data to A^L files, and (d) the capability to ex- 
tend data base sizes. It should be noted that the file access 
subroutine packege was built as a direct extension onto the file 
system capability which already had extended the APL data base 
functions. For example, these capabilities allowed a major pro- 
ject of instructional usage in information retrieval systems for 
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the School of Library Science* The IR Project was able to develop 
a system of automated abstracting and retrieval oh the APL file 
sys'teia. Tiie system also provided instruction for g^duate library 
science students in thje areas 'of automated library retrieval sys- ^ 

.i ; t J - 

"terns. Experience^ v^rith this IR function indicates that any system 
of computers in education must provide the capability for a well- 
defined file •system with variable structuring andj flexible manipu-. 
lation. 

5.2.2 Data Base Manipulation on the Coursewriter System ^ 

Specific 'requirements were "found for manipulation of large 

<•' / if' 

files on a regular b^si£. Because of the volumes of data collected 

in CAI research, highly efficient programming was needed to allow 

***** 

maximization of program execution times and therefore operations. ' 
One of the major problems in all computer systems , hot Just in • 
0^&$}X'3fct& ^ys temg ^ is the bottleneck^) f input a^d output functions. 
That is /program execution iz no faster than the ti*ne it takes to 
pass data through ti 9 the "throughput" time* However, throughput 
is primarily a function of /the time for input and output. 0 Through- 
put improvements were accelerated by programming internal execution 
speeds »so that programs approached the I/O boundary conditions 
while still maximizing execution efficiency. 
5.2.2.1 Sort and Merge 

An internal, sequence-sensitive, bi-directional sort was 
'developed to take advantage of the fact that CAI records and data 
are not randomly ordered, but are rather more often slightly out 
of sequence. This occurs because data are recorded for students 
within specific courses at approximately the same time and dates. 



The bi-directional sort locates the correct sequence position of 
a currently encountered , outr-^f -sequence record in no more than 
eight comparisons for any 100 response records. The software 
management allows blocking; of records in groups of 100. In addi- . 
.tion to the fast location finding of the sort, no data are imme- 
cjiatelyvswapped because a fable of locations is kept and accessed 
for addressing -the individual records. That is, intermediate data 
swaps have been eliminated by indirect data examinations via a bi- 
directional linked-index listing procedure. 

In addition to the high speed sort f it was also necessary 
to develop specialized merging programs which allowed the ordering 
ot data by specific request. For example , it might be desirable; 
to have re6ords only for a gi^en course, records for a given course 

within a specific time period, or records only for a segment of a 

-id- 
course. For computational efficacy, it is expedient to block 

these specific records together ana eliminate others. The merge 

and sort programs, when specifically oriented toward the type of 

semiordered data accumulated in CAI , fulfill these requirements 

mostj efficiently . 

5.2.13 Instructional Support feystem (IjSS) 

j A major software -developrnentpfro ject undertaken at FSU was 
the redesign of the FSU data management system and IBM CAI system 
such that they operated under one complete support and operating 
system. That is r either system could be controlled under one set 

y 

of operating procedures. This integration of the two programming 
systems allowed greater optimization of computing hardware utili- 
zation, and a significant improvement in computer, operations . 
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5.2.4 Hanagement Operational Procedures 

Prime sources of maximizing the utility of any instructional 
management system are , the operations procedures and scheduling • 
FSU has nade a major effort to ensure ( that the human component of 
the instructional management system is well integrated with the 
software and hardware components. Software programs which allow 
editing 'and preparation of data for d&ta analysis were developed , 

along with complete user documentati^h. For researchers, the 

/ 

special need ' for manipulating data, for use in statistical software 
packages was taken into account. / Given the variety of data whiqh 
may be collected on a sophisticated CAI or CMI system and the 
varieties of research designs for analysis requirements , flexible - 
and powerful control of data must be available for researchers. 

5.2.5 Future , . 

In the future, investigations should focus on the develop- 

f 

ment of neW technologies for recording vast amounts of data during 
computer-aided and computer-managed instruction activities. As • 
instruction becomes more individualized and personalized, managing 
and monitoring systems must be expanded. 
5.3 Data Analysis Systems 

The business of all instructional research is analysis and 
interpretation of data gathered. In order for the data to be use- 
ful, however , the CAI/CMJ system must be well integrated with data 
analysis systems. The thrust of the FSU effort with concern to 
data analysis has been in the direction of providing new user capa- 
bilities for analysis of data. 
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5.3.1 Online Versus Offline Systems 

Many CAI researchers have suggested the need for data 
reduction and presentation in an online mode/ with data output 
to the researcher as students proceed through instructional treat- 
ments on display devices. The offline mode allows gathering of 
data on intermediate storage media which are accessible at later 
times for data reduction and presentation. The offline mode has 
been primarily emphasized at FSU. The advantages of offline 
data recording are in ease of operations and flexibility with 
which raw data can then be transmitted to data analysis systems. 
^ Many manual operations necessary for owline data gathering are 
bypassed with offline functions. 

5.3.2 Report Generation f 

Both the EDIT and DATA PREPARATION programs provide punched 
^utput of data collected during CAI coursewriter activities for 
^use on various computer system statistical packages* In additiq>p, 
two programs were developed at FSU which provide summaries in t&e 
form of printed output reports of CAI data. , The ITEM ANALYSIS 
f SUMMARY program provides summary statistics such as the percentage 
of Students who answered an item correctly and the average latency 
time for the iterft. The DETAILED PRINT program prints, a comprehen- 
sive detailed description of a student^or subject's record. Infor- 
mation such as course name, date, EPID, MID, studeht response, 
response latency, and value of counters and switches is provided. 
This program has a gr »at deal of flexibility through the use of 
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J2 options for output specification to select or suppress specific 
in formation elements . 
5.3.3 Future 

If, in the future, the trend continues toward more power- 
ful general purpose and algorithmic-type languages for CAI , then 

V & 
more general and ^powerful report generation systems will have to 

be developed with ^ome loss in execution optimization for the gain 

of power and flexibility. 
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G . 0 D evelopment of Humary Resources 

A corollar'v goal of the ONR Themis project was an explicit 
commitment to develop a cadre of instructional researchers with 
ai(~Tnterest in conouter-based-irr^ Dtllring the pro- 

ject at FSU, a- large number of these students, at both the Ph.D. 
and Piasters Degree levels, have participated extensively in the 
research activities. As a review of the reference list will reveal, 
these individuals .participated as researchers and as authors. 
Most encouragingly, these The Tiis-related graduate students contin- 
ued the "themes pursued under Themis. Explicitly, One can identify 
the continuing interest in and widespread developments of computer- 
managed instructional systems.. A brief review ijbf gome of these on- 
going research activities/ and then a listing of all of the parti- 
cipants, will document this proposition. ' 
6 . 1 Outstanding Computer-Managed Instructional Research Projects 

The following individuals have assumed significant roles in 
other institutions or organizations, and are continuing the themes 
developed under Themis. 

1. Paul Gallagher, Ph.D., Assistant Dean for Instructional 
Systems, Florida International University. 

2. Lorraine Gay, Ph.D., Assistant Professor of Education, 
Florida International University. 

> 

These researchers are responsible for the development of 
an individualized performance-based teacher education program. 
Within this activity they hatfe specific responsiblities for the 
development of a computer information management system which 
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supports the essential ruortitoring and control functions necessary 
for this innovative model. For those research training endeavors 
in colleges of education in the United States, the Florida Inter- 
national University performance-based instructional program is con- 
sidered to be further ahead and to be "breaking new ground" at this 
time. ' * 

3. Barbara Leherissey McComb, Ph.D., Department of Engineering 
.Psychology, McDonnell Douglas Corporation. 

For the past y^ar this researcher has been responsible ,for 
formulating the conceptual ideas required to implement the instruc- 
tional strategies subsystem of the* Advanced Instructional System 

f 

at Lowry Air Force Base. McDonnell Douglas Corporation has been 
th(Tyfcuccessful bidder and is now prodeeding in tn5" development of 
the instructional. approach. Again, the adaptive models to be used 
will Have heavy emphasis on the management of the training process* 

4. Gary Lipe , Ph.D. , Director of the CAI Center at Texas 
Christian University. 

Texas Christian University has been pursuing individualized 
teacher education, emphasizing units on educational* technology. 
The TCU Center has developed unique computer- and media-related 
presentations which are having significant impact in building com- 
petencies for future teachers. . . % 

5. John Hedl, Ph.D., Chairman of the Educational Psychology 
Department, University of Texas .Medical ►School , Allied 
Arts Medical Center. 

v 

ERJ.C 

/ 
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This researcher is developing a computer-managed instruc- 
tional approach, to allied medical manpower development. In addition 
to bqing competency based, the computer approach will emphasize 
professional compp^ency profiles and a system for retraining in 
the* future . 

5. Arthur King, Ph.D. , Director of Instructional Systems, 
Dental School, University of Florida. 

This researcher has developed a computer-managed instruction 
approach t<q£all of the medical science requirements found in dental 
schools. This system is now operating and is undergoing extensive 
formative evaluation. 

6. Kenneth Majer, Ph.D., Assistant Professor of Educational 
Psychology, Indiana University. 

This researcher h^s developed an individualized graduate 
assistant teacher training system using systematic principles and 
computer management. This system provides instruction to graduate 
assistants from departments throughout this large university. The 
developments ar^ considered so exemplary as to have Won an award 
during iihis pasyc year. * ■ 

7. Dr. Harold O.'Neil, Ph.D., Co-director of the CA3T Center, 
University of Texas. 

This researcher is providing the intellectual leadership 
and management of a large research program in CAT and CMI at Texas. 
Special efforts are focusing on the "nature of learner control of 
instruction and affective processes found in anxiety and curiosity, 
in addition to the develbpment of a CMI approach to instructional 
science for undergraduate and graduate students. 
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While v other individuals can be cited , these give a repre- 
sentative sample of the gr achate students who have used ^heir 
training and substantive themes developed under Themis in order 
to pursue profitable careers throughout the\United States. 
6 . 2 Themis Associated Faculty and Students ' \ 

During the project , some faculty merabers\moved on to more 
responsible positions in other organizations* Th^se are as follows 

1. Bob Brown, Director of the CAI Laboratory, University of 
Iowa, Iowa City, Iowa, 

2. Henry Lippert, Instructional Coordinator for CA3\,-Army 
Medical Training School, ?orty Sara Houston, Houstoi^, Texas, 

* 3. Walter "Dick, Director of Plamning and Head of the Instruc-. 

tional Systems program, Florida Sta±e University. 

The following graduate students also participated in the 
Themis project. Their present job titles and research activities 
are offered for the reader's analysis .» Readers will observe, that 
these former students have continued their interests in instrucr 
tional and management systems as well as -the whole ar^va of training 

1. Wallace E. Bell, Ph.D., is a Coordinator of Information 
Management Systems for Evaluation and Research, Florida 
Department of , Education, -Tallahassee, Florida. 

2. Allen A. Blair, M.S., District School Superintendent , State 
' of Washington, is implementing individualized instruction 

in his' school district. 

3. '. V7alt Blomquist, M.S. , Coordinator of CAI and Instructor of 

Speech, Brevard Community College, Brevard County, Florida, 
is? ^ an innovator in computer activities . 



Thomas Dunn, Ph.D. , Assistant Professor of Education at 
Toledo University Ohio, is very active in the performance- 
based computerized teacher education program. 
Edwin Durall, Ph.D., Director of the Department of Research 
and Evaluation, Myrtle Grove Instructional Center, Escambia 
County School Board, Pensacola, Florida, is active in 
computer-supported evaluation . 

Paul Geisert, Ph.D., Assistant Prof essor, University of 
Wyoming, is developing individualized courses in science 
education, and is implementing a competency-based Masters 
^and Ph.D. program. 

Fernando Gonzalez, Ph.D. , Assistant Professor of Psychology 
at Florida International University , is active in computer- 
based testing. 

Darol Graham, ABD, . Research Associate in the FSU. 
Habilitative Sciences project to individualize teacher 
training, is developing a computerized management system. 
William R* Halstead, Ph.D., Director of Higher Education 
Planning, Department of Education, State of Florida, is / 
developing a computerized data system for planning. 
Nancy K. Hagerty, Ph.D., Assistant Professor, Department of 
Continuing Education, University of British Columbia, is 
working with inservice teacher training particularly using 
performance-based teacher education. 

Robert Kandelsman, Ph.D., Director of Financial Aid, Miami- 
Dade Junior College, developed media learning systems. 



Wallace Hannum, Ph.D. , Research Associate, Center for 
Educational Technology, FSU, is developing workshops in 
instructional design for foreign students. 
Edward Harris, Ph.D./ Assistant Professor, Department of 
Educational Administration, Florida State University, is 
teaching systems analysis and computer techniques for ad- 
ministrators. 

William Harvey, Ph.D., Assistant Professor of Educational 
Technology, University of Southern California, is developing 
teaching-management simulations and games* 
Karl -W, Heiner, Ph.D., Assistant Professor of Education, 
University of California, San Francisco's Medical School, 
is teaching in the allied health professions, program and 
developing computer-based evaluation systems. / 
Darlene Heinrich, ABD, is currently investigating anxiety 
and curiosity within systematic * learning situations, 
Edward N. Hobson, Ph.D., Program Director,. National Labora- 
tory for Higher Education, Durham, North Carolina, is 
directing the junior college division that is preparing 
materials and activities to train professors in the use of 
systems approach and planning. 

Robert M. Hogan, M.S., is, completing a degree in mathema- 
tical psychology at the University of Colorado.^ 

J . y ■ 

t ' \ 

Thomas James, Ph.D., Research Associate, Center for Educa- 
tional Technology, is assisting in the development of 
workshops in instructional design, and is developing an 
information retrieval system for the CET library. 



Barbara F. Johnson , . ABD , Assistant Director of the > 
Computer Applications Laboratory at FSU, is involved in 
computer applications throughout the university . .;. ' 

John B. Keats, Ph.D. , Associate Professor of Statistics, 
Southwest Louisiana State University, is pursuing statis- 
tical algorithms for management. 

Cyrur Kirshner, M.S. , Associate Professor o'f Vocational 

Education, Los Angeles State University, is completing his 

dissertation in applying a systems approach to the devel- 
.» * 

^ > ' , s . * 

dpment of instructional materials for: vocational education. 

Dewey Kribs, ABD, Research Psychologist, Naval Personnel 

and Training Research Laboratory',. San Diego, will implement 

advanced instructional systems concepts. 

Raymond F.Latta, Ph.D., Associate Professor, Western 

'Washington State University, is working to train educational 

administrators in systematic planning analysis techniques 

based on computing \sys terns . > 

Robert M. Lawler, PhuD. , Assistant Professor, Allied Health 
Program, University of - California at San Francisco, is 
^veloping a performance -based training program with compu- 
ter support. * 

Michael H. Moncrief, Ph.D., Research Associate, Southwest 
Regional Education Laboratory, Los Angeles, is preparing 
individualized learning systems. . 
Merle Morgan, Ph.D., Director of Innovative Instructional 
Design, St. Petersburg Junibr College, ist -developing new 
teaching learning systems. .* ■ 



/ James Papay, Ph.D., Research Associate, Dallas ' Public 
School System, is conducting research and evaluation on 
Title 1 Projects in Dallas. 

William' Proctor, Ph.D., President of Flagler College, 
St. Augustine , Florida, is pursuing "a systems approach to 
college administration . , , 

Gail T„ Rayner, Ph'.D. ,~ Research Associate, Training D^Lyi : - 

* * * » 

sion, Center of Educational Technology, is developing 
.training activities for students* from- the Organization of 
Araerican States. ^ , 

Lee Rivers, PhiD., Staff Psychologist, HR£ Singer, is 
specializing in the development of instructional materials 
•for training systems. - ' 

Worth Scanland, Ph.D. , e/d^cational s^e^alist with Naval 

Training Command, is specializing in educational technology 

v * c »«• 

^or Naval training systems. ., . • • 

ciharles /isson, ABD, Executive** Director of Management and 

1 A • * * * • 

. Information Services , Colorado State Department of Education 

^is in charge' of bi^3^e;t control, PPBS computer/serviqes i 

and internal business management. 

Ben Stevens, M.S., Research Assistant, University of . 
Florid* Medical School, is assisting in the development of 
instructional materials. \ n 
r Peter Stoychef, M.S. , Research Associate, School of 
Dentistry, Ohio State University, is -pursuing the system- 
^^Stic design and evaluation of CAI. 
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Susan Taylor, A3D, is currently investigating adaptive 
instructional models at the FSU Computer Applications 
Laboratory, 

David B. Thomas, Ph.D., Research Associate, Department of 
Data Processing, State of Virginia, is reorganizing the 
State's use of computers in educatdon for better management 
of instruction. 

Nelson Towle, PhuD. , Research Associate at CCSI, Florida 
State University; is helping faculty develop CAI/CMI units. 
William Whaley, ABD, Project Director, Department of 
Education, State of Florida, Program Services Division, is 
developing dif f jrentiated staffing patterns with manage- 
ment support by computers . 
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7 . 0 List of Technical Reports , Technical Memos , Systems Memos, 
a nd Working Papers ^ 
Between 1967 and the close of the FSU Themis Project , 
approximately' 100 technical documents were written and distributed 
as a* result of research activities connected with the project. 
These technical reports, technical memos, systems memos, and 

v/orking papers are listed below by type of paper, number of paper, 

y 

author, title, and date. 
d 

T echnical reports 

6-. O'Neil, II. Effects of ^stress on state anxiety and perfor- 
mance *iri computer-assisted learning* 1969. 

^7. Spielberger, C« The effects oS anxiety on computer-assisted 
learninn, 1969. 



8. Jlobson, 3, Empirical development of a computer-managed 



instruction . system for the Florida State University model for the 
preparation of elementary school teachers , ^1969. 

11. Hagerty, K. Development and implementation of a computfer- 
mahaged instruction system in, graduate training y 1910. 

12. Gallagher, An investigation of instructional ' treatments 
and learner characteristics in a computer-managed instruction 
course, 197 0. 

P 13. Scanland, W. An investigation of the relative effectiveness 
of two methods of instruction-, including computer-assisted instruc- 
tion', ac techniques for changing the parental attitudes of 'Negro 
adults, 1970. 

14. Spielberger, C.., O'Neil, H. , & Hansen, D. Anxiety, drive ~ 
theory, and computer-assisted learning, 1970. ■ V 
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15. Lipe, J,, The development and implementation of a model 
for the design of individualized instruction at the* university 
leva I, 19 70. * ' * f 

16. King; A. An application 'of simulation techniques to an 
innovative teacher training program* 1970. 

• 17. Harvey, W. A study of the cognitive and affective out- 
comes of a collegiate ^science learning game* 1970, 

18... Gay, L. Temporal position of reviews and its effect on 
the retention of mathematical rules* 1971. < 

19. Lawler/ M: An investigation of selected instructional 
strategies in an undergraduate computer-managed instruction 
^course y 1971. 

21. Hedl, J. An evaluation of a computer-bctsed intelligence 

I 

test, 1971. " ■ 

22. Dunn, T. The .effects of various review paradigms on 
performance in an individualized > computer-managed undergraduate 
course* 19 71. * * 

) r , 

23. Leherissey, B. The effects of stimulating vtate epistemic 
'curiosity on statTe' anxiety and performance in a comptex computer- 
assisted learning task* 1971. 

26. Motley, D. An on-line computer-managed introduction . to 

■/». .■■ 

indexing : An individualized multimedia instructional package com 
l-sared to the traditional method* nine hours of teacher-group 
contact* 1972. .. ? . M 
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27. Rivers, L. Development and assessment of ar\ adaptive , 

t 

r/f.yabei< utilising, regression analysis techniques fo? the presen- 

' ' ? 

tail on of tnntruc liorx^ : oia computer * 15 

Technical Ilemos 



Lgf?2. ' * J 



1. pick, W. ' Implementation of CAI at^Florida State IMiversity* 
1969. ^ . 

>^ 2. Hansen, D. ' Current issues in CAI* ,1969. 

4. Ehlers, W. CAI in social work, 1969. 

5. Dick, W. & Latta, R. Comparative effects of ability and % 
preceniati mode in computer-assisted instruction and programmed \ 
inct ruction * 1969. . 

3. Hansen, D. & Dick W. The data world of CAI * 1969, 

i * 

10. Hansen, D. & Harvtiy, W. ' Impact of CAI on classroom teachers 

1969. ' " ^ 

11.. Hansen, D. Development processes in CAI problems* tech- 

i ■ r 

* * ■ ■ 

niques* and implications * 1969* - - . 

12* Gay L. An investigation into the' differential effective- 
ness for males and' females of three CAI treatments on delayed re- 
tention of mathematical concepts* 1969. 

14. Lee, R. Information structure in military history : An 
application of .computer-assisted instruction* 1970* N 
\15. Hanseh, D. The role of computers in education during the 
■ f 70 r c* 1970. / \ 
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16. Adair, C. ^-Hansen, D\ , Rayner, G„., & Agarwal, A. The ~ 

hi'hfioiot' of l.^uclirr*:: involved in Lh)o simulated inquiry environ- 
ments* .1 social i: imu'Latiprv game and a CAT-based information 
rnlrybwl eye tern, 19-70,. ** * w 

18. 'Sheldon, J. Computer -as sis ted instruction in engineering, 
dynamics , 1970. 

■ 19. Bruce, D . & Papay, J. The primacy effect of single-trial 
free recall, 1970. ' ■' % 

20. Leherissey, B., O'Neil, H., & Hansen, D, Effects of memory 
support on state anxiety and performance in computer-assisted ' f 
learning, 1970. • j \ * 

22. 4 Jahoda, G. &" Foos , F. The development of an on-line 
searched oobrrlinate index for use in teaching and research, 1970. 

23, Kromhout 6 ', 0. , Hansen, D., & Schv/arz , G. Conference on * 1 
computers in undergraduate science education : A computer-assisted 

anfl managed course im-physical sciehv&s , 1970. 

f 

124. Geisert, P. A comparison of the effects of information'- 
mapped learning materials and traditional materials on the learning 
of concepts vig. the printed page and computer cathode ray tube, 1970 

25. Graham, D • , Schwarz, G. , & Hansen, D. Multimedia Simula- 
tion of laboratory experiments in a basic physics lesson on 
magnetism, 1970. i ' ■'/ ■ . 

27-. Merrill, P. Task analysis-- An information processing 
approach, 1971. 

23.. Dunn, T, , Lushene, R. , & O'Neil-, H. The complete, auto- 
mation of the M^jj^es ota ' Multiphasic Persality Inventory and a 
study of its response latencies , 1970. » 
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30. 'Hansen, D. , Hedl, J., & O'Neil, H. Review 9/ automated 
La tin : t .19 71. 

31. Thomas, D. applications of simulation in the edu<hx^ 
tional environment * 1971. 

32. Dick, W. & Gallagher, P. Systems concepts and computer-' 
managed instruction: An implementation and validation study* 
1971. 

33. Rappaport, E. The effects of trait and dogmatism on state 
anxiety during computer-assisted learning* 1971. 

34. Leherissey, B. The development of a measure of state 

* 

epidemic curiosity *■ 1971. 

35. Hansen, D. & Johnson, B. Measurement' techniques for 
individualized instruction- in CAI* *1971. 

36. Brov/n, 3., Hannum, W. , & Dick, W. An investigation of 
the effects of two types of ins true tiond'l terminals in computer* 
managed instruction, 1971. j 

37. Brown, B. & O'Neil, H. Computer terminal selectio^ Some 
instructional and psychological implications * 1971. c 

38. Hansen, D..& Johnson, B. CAI myths that' need to be destroyed 
and GAL myths that we ought to create* 197}... 

3-9- — S^anune r 1 i n- r — L-. — A^-f&astbility study of tutorial type 



computer-assisted instruction in selected topics in high school 
chemi s try * 19 71. 

40. Jahoda, G. & Foos, F. The use of an on-litye searched and 
printed coordinate index in teaching* 1971. 
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41. Leherissey, 3.,0^Neil, H., & Hansen, D> Effect of .anxiety * 
resvoKr-: \-:ode, and subject matter familiarity on achievement in 
compute^-acsis ted learning, 1JJ71. 

42. Leherissey, B., O'Neil, H., Heinrich, D. , & Hansen, D. 
Hffrat 'sf anxiety, response mode, subject matter familiarity, and 
Lnai^i>'. t / time on achievement in compu ber-a&sieted learnina y 1971. 
\^Jr%. Tobias, S. A history of an individualized- ins true JionaV 
pvogvan of oarying familiarity to college students , 1972. 

44. Tobias, S. Review of the response mode issue, 1972. 

45. Mejrill, P. & Duchastel, P. The effects of behavioral 
obj actives on learning : A revi.ew of empirical studies, 1972. 

46. Tovle, N. & Merrill, P. Effects of anxiety typ.e and -item 
difficulty sequencing on mathematics aptitude test performance* 

47. .Merrill, P. & Towle, N. The effects of the 'availability of 
objectives on performance in a computer-managed graduate course* 
1972. _ • 

48. - Lushene, R. , O'Neil ^ H. , & Dunn, T. Equivalent validity 
of a completely computerized MMPI, 1972. 

49. Tobias, S.& Hedl, J. Test anxiety: 1 ] Situationally specific 
or ge>ieral, 1972. x_ j 

52 Tobias , S . Distraction, response mode* \nxiety * arid 
achievement from GAI,13~}2J 

54. Hansen, D. , Manning, P*V & Johnson, B.^ A systems approach 
i'o learning simulations* 1972. A 



"56. Tobias, S,, Hedl, J., & Towle, W. Latency and test 
anxiety , 1972. ; 

57. .Tobias, S. & Duchastel, P. Behavioral objectives , sequence* 
and aptitude treatment interactions in CAT'* 1972. 



58. Tobias, S. & Leftwich, M. Study habits and test (^^ie'ty, 
1972. • 

59. Merrill, P. , - Steve ,' M. , Kalisch, S . , & Towle , N , The 

interactive effects of the availability of objectives and/or rules 
on computer-based learning ; 'A replication* 1972. 

60. Dunn, T. & Hansen, D. . teaming by graphics : translating 

' ■ • * ' «V *\/ V. .. . .. . ■ 

verbal information inio graphic network formats* 1972. - 

61. M.errill, P* & Tov/le,* ti. Interaction of abilities, tend 
anxiety with availability of objectives atid/or test items on 
computer-based task performance , 1972. 

64 n Kromhout, 0. \ Effect of computer tutorial review ,lesso'ns on 
exam performance Jin introductory college physics 3 1972. 

66. Duchastel, P. ' Incidental and relevant learning with 
instructional objectives , 1972. ■ " . 

Systems Memos . 

m i - - - , , \ ■ 

1. OVNeil, H* , Hansen, D • , Freed, 3., Wester, E., Hogshead, G. 
& Robinson, ,W. Documentation for the edit and data preparation 
programs, 1970. 

2. O'Nail, H. , Papay, S., & Hansen, D. Supplementary documen- 
tation of Coursewriter II functions y 1970. * . 

3. £ippert, H. , Harris , E. , Thomas, D., & McMurchie, T. . 
of APL/1S00 functions, 1970.' 
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4. Lippert, H. & Harris, D. APL: An alternative t<? the 
mult: language environment for education, 1970. 

5. Freed, B. , OVNeil, H. , Lauffer, L. , & Lushene, R. 
teleprocessing: A DEC 680 interfaced, to an IBM 1500, 1970. 

6. Hansen, D. , Papay, J. f O'Neil, H.< & Pawner, D. Human- 
.io^putcr interactions involved in analysis of CAI data, 1-970. 

9. Kribs, D . , Wright, B., & Reynolds, E. /FOCAL manual for 
■'//>. Coding <>n the TSll/B 'system, 1970 • 

10. McMurchie, T. & Thomas, D. APL/1500 file access sub- 
routine package, 1971. 

11. McMurchie, 'T. & Thomas, D. Manual of APL/1500 functions: 
Systems functions , 1971. 

12. Thomas, D., Merrill, iP. , & Hansen, D. STATSIM: Exercises- 
in statistics, 1971. 

13. • McMurchie, T. & Krueger, S". A programming language/ 1 500 
(APl/lbOQ) operator's guide, 1971. 

14. McMurchie, T. & Thomas, D. Manual of APL/1500 functions: 
Systems functions , 1971. -, 

15. McMurchie, T. Operation of offline programs for generating 
and maintaining the APL/1500 system, 1972. 

Working Paper 

11. Towle, N. Effects of the memorization of rule statements • 
on performance , retention, and transfer in a computer-based 
learning task, 1973. 

12. Taylor, Susan S. A model for the design and evaluation of 
an adaptive drill and practice instructional program, 1973. 
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